Dynamical Spectrograms That Can Be Perceived as Visual Gestures

A new system for speech visualisation, has been implemented to allow deaf and hearing-impaired people to understand verbal information over channels such as the ordinary public telephone system. Incorporating a computational model of the human ear, the system converts incoming sounds into a sequence of animated images, which show the temporal variations of the spectral pattern of the input sound in real-time and which are perceived like visual gestures. Preliminary results from forced-choice tests with 28 human subjects are reported, using a sequence of 2to 4-word sets. To demonstrate the language independence of this approach, some of these were taken from 4 very different languages English, Persian, French and Czech. The results show high levels of recognition after only 10 learning trials (typical mean scores of 50-85%, where zero represents chance expectation), and encourage further investigation.